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THE  DEVELOPMENT  OF  THE  ATC  SELECTION  BATTERY:  A NEW  PROCEDURE  TO  MAKE 
MAXIMUM  USE  OF  AVAILABLE  INFORMATION  WHEN  CORRECTING  CORRELATIONS  FOR 
RESTRICTION  IN  RANGE  DUE  TO  SELECTION 


Introduction. 

To  develop  or  update  a teat  battery  used  for  selecting  personnel,  two 
very  important  steps  must  be  completed.  First,  the  most  valid  tests  must 
be  chosen,  and  second,  a weighting  system  must  be  devised  which  will 
combine  these  tests  into  a composite  that  yields  a maximum  validity 
coefficient.  In  order  to  do  this  all  tests  under  consideration  are 
intercorrelated  with  each  other  and  correlated  with  a specified  criterion 
of  Job  success.  These  correlations  are  used  to  regress  the  test  scores  on 
the  job  success  criterion  and  the  coefficients  from  the  regression  analysis 
are  then  used  to  determine  which  tests  should  be  included  in  or  deleted 
from  the  battery  and  what  the  relative  weights  should  be  for  each  test. 
These  weighted  test  scores  are  then  combined  to  form  the  composite  3core 
which  is  used  for  selection. 

In  the  selection  of  air  traffic  controllers,  a five-test  selection 
battery  is  currently  given  to  applicants,  each  test  score  is  weighted,  and 
the  five  weighted  scores  are  combined  to  form  a composite  which  Is  used  to 
select  candidates  for  Air  Traffic  Control  (ATC)  training.  This  test 
battery  is  in  the  process' of  being  revised,  and  several  new  selection  tests 
have  been  developed  which  could  replace  part  or  all  of  the  existing 
five-test  battery.  To  evaluate  these  new  tests  and  compare  them  with  the 
existing  battery,  they  were  administered  to  7,000  ATC  applicants  along  with 
the  existing  five-test  battery.  The  applicant  scores  on  the  five  existing 
tests  and  the  new  tests  were  then  correlated  to  see  how  much  overlap 
existed  between  them. 

In  order  to  determine  the  utility  of  the  tests,  both  old  and  new,  it 
was  necessary  to  correlate  them  with  some  criterion  measure  of  Job  success. 
Unfortunately,  job  success  measures  are  available  only  for  those 
individuals  selected  to  be  controllers,  and  this  selection  is  based  on 
scores  only  on  the  five  current  selection  tests.  An  important  factor 
influencing  the  size  of  correlation  coefficients  between  a test  and  the 
criterion  Is  the  range  of  scores  available  on  the  tests  and  on  the 
criterion.  Since  information  about  the  Job  success  criterion  is  available 
only  for  the  ATC  applicants  who  have  been  selected  for  employment,  only  the 
upper  range  of  scores  is  available  on  the  criterion.  Because  of  this 
restriction  in  range,  the  correlations  between  the  current  selection  test 
scores  and  the  Job  success  criterion  will  be  spuriously  low.  This 
situation  is  illustrated  in  Figure  1. 

The  new  tests  being  considered  to  replace  part  or  all  of  the  existing 
test  battery  will  have  a larger  range  and  variance  in  the  selected  group 
than  the  five  tests  actually  used  for  selection.  In  fact,  the  range  and 
variance  will  be  restricted  only  to  the  extent  that  the  new  tests  correlate 
with  the  old  tests,  and  will  be  as  restricted  as  the  old  tests  only  if  this 
correlation  is  1.0.  Because  of  this  differential  restriction  in  range,  the 
new  tests  will  correlate  higher  with  the  Job  success  criterion  in  the 
seleoted  group  than  will  the  old  tests. 


Figure  1.  The  effect  of  restricted  range  on  a correlation 

coefficient.  Subjects  in  the  smaller  box  represent 
the  selected  group.  The  unrestricted  correlation  of 
the  two  variables  is  .88,  and  the  restricted  is  .15. 


To  adjust  for  this  spurious  result,  the  correlations  with  the  job 
success  criterion  must  be  corrected  for  restriction  in  range  to  assess  the 
validity  of  the  tests  used  for  selection  and  to  determine  now  the  current 
tests  used  for  selection  compare  with  the  new  tests.  The  correction  must 
take  place  prior  to  performance  of  regression  analyses:  otherwise,  the  new 
tests  will  appear  superior  to  the  current  tests  because  of  nothing  more 
than  a statistical  artifact.  This  also  means  that,  when  corrected,  the  new 
test  correlations  with  the  criterion  will  generally  increase  less  than  the 
old  test  correlations. 

Since  a composite  score  is  used  for  selection  of  ATC  trainees,  and  the 
five  tests  in  the  existing  battery  are  not  given  equal  weights,  some  tests 
in  the  battery  contribute  more  to  the  composite  than  others.  Because  of 
this  differential  contribution  to  the  composite,  some  of  the  five  tests 
which  form  the  composite  will  be  more  restricted  in  range  than  others. 
Consequently,  the  correlations  for  some  tests  which  form  the  composite, 
when  corrected  for  restriction  in  range,  will  Increase  more  than  others, 
and  the  amount  of  increase  will  be  proportional  to  the  amount  of 
restriction  in  the  variance  for  each  particular  test. 

Equal  Employment  Opportunity  Commission  (EEOC)  Guidelines  state  that 
tests  used  for  personnel  selection  must  be  demonstrated  to  be  valid 
predictors  of  job  success,  and  the  magnitude  of  the  validity  coefficient 
must  be  both  "practically  and  statistically  significant"  (3).  The 
spuriously  low  correlation  coefficient  due  to  selection,  then,  becomes  a 
very  important  legal  issue  in  addition  to  its  importance  in  assessing  the 
value  of  new  selection  tests.  Numerous  litigations  have  occurred  as  a 
result  of  this  problem,  several  of  which  related  to  the  accuracy  of  the 
methods  employed  in  correcting  the  validity  coefficients  for  restriction  in 
range  (1). 
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There  are  two  major  statistical  formulas  which  have  been  developed  to 
correct  the  correlation  of  a test  and  a job  success  criterion.  For  the 
purposes  of  this  study , the  following  notation  will  be  used  for  all 
formulas: 


x = the  current  selection  composite  score 
y = the  new  test,  or  one  of  the  five  components 
of  the  current  test  battery 
z = the  job  success  criterion 
RR  = the  unrestricted  correlation  of  the  variable 
subscripted 

SS  = the  unrestricted  standard  deviation  of  the 
variable  subscripted 

R = the  restricted  correlation  of  the  variable 
subscripted 

S = the  restricted  standard  deviation  of  the 
variable  subscripted 

Both  major  formulas  estimate  the  value  of  RRyz  based  on  the  information 
available  on  the  restricted  group:  Rxy,  Rxz,  Ryz,  Sx,  Sy,  and  Sz.  They 
differ  in  their  assumptions  about  information  available  on  the  unrestricted 
group. 


The  first  formula  (5),  Thorndike's  formula  7 case  III  (hereafter 
referred  to  as  T7) , assumes  that  only  SSx  is  available  for  the  unrestricted 
group  and  uses  the  ratio  SSx/Sx  and  the  restricted  correlations  to  estimate 
RRxy , RRxz,  SSy  and  SSz.  These  estimates  in  turn  are  used  to  estimate 
RRyz.  The  second  major  formula  (4) , Gulliksen's  formula  37  (hereafter 
referred  to  as  G37),  assumes  that  only  SSy  is  available  on  the  unrestricted 
group  and  uses  SSy-Sy  and  the  restricted  correlations  and  variances  to 
estimate  RRxy,  RRxz,  SSx,  and  SSz.  These  also  are  used  to  estimate  RRyz, 
which  is,  of  course,  the  desired  unrestricted  correlation  of  the  test  and 
the  job  success  criterion. 


The  problem  in  using  either  of  these  formulas  for  the  ATC  selection 
situation  is  that  both  T*  and  G37  require  making  estimates  of  either  SSx  or 
SSy  and  RRxy,  when  this  unrestricted  information  is  actually  available  from 


for  correcting  for  restriction  in  range  using  available  unrestricted 
values.  In  the  two  formulas  already  developed,  estimates  of  SSz  and  RRxz 
only  are  required  to  estimate  RRyz.  In  order  to  make  maximum  use  of  the 
unrestricted  information,  two  formulas  were  derived  by  the  first  author  of 
this  paper.  The  first  formula  (hereafter  referred  to  as  B1)  uses  SSx  to 
derive  estimates  of  SSz  and  RRxz.  The  second  formula  (hereafter  referred 
to  as  B2)  uses  SSy  to  derive  estimates  of  these  variables.  In  both 


to  as  B2)  uses  SSy  to  derive  estimates  or  these  variables.  In  both 
formulas,  the  estimates,  along  with  the  actual  unrestricted  values  of  RRxy 
and  either  Sx  or  Sy,  were  used  in  conjunction  with  restricted  correlations 
to  estimate  RRyz.  The  four  formulas  were  compared  both  mathematically  and 
by  using  Monte  Carlo  techniques  to  determine  which  can  be  most  accurate  in 
estimating  RRyz  across  different  selection  ratios  and  different  correlation 
values. 


ttethola- 

Following  Gulliksen's  (4)  schema  for  derivation  of  the  correction 
formulas,  three  assumptions  were  employed,  where  upper  case  and  lower  case 
letters  represent  unrestricted  and  restricted  variables  respectively  and 
x = the  test  used  for  selection,  y = the  new  test  being  assessed  and 
z = the  success  criterion. 


t 
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Assumption  1 . The  slopes  of  the  regressions  of  the  new  test  and  the 
criterion  used  for  selection  are  not  affected  by  selection. 


Rxy  Si  = RRxy  £Sy. 

Sx  SSx 

(1) 

Rxz  Sz  = RRxz  SSz 
Sx  SSx 


Assumption  2.  The  error  made  in  estimating  the  new  test  scores  and 
the  criterion  from  the  selection  test  scores  is  not  affected  by  selection. 


2 2 2 2 

Sy  (1  - Rxy  ) = SSy  (1  - RRxy  ) 

2 2 2 2 

Sz  ( 1 - Rxz  ) = SSz  ( 1 - RRxz  ) 


Assumption  3.  The  partial  correlation  between  the  new  test  and  the 
criterion  is  not  affected  by  selection. 

Rvz  - RxyRxz = aMs_-_R-RmMx3- (3) 

~\J ( 1 - Rxy2)(1  - Rxz2)  { 1 - RRxy2)(1  - RRxz2) 


Based  on  assumptions  1 through  3i  derivation  of  the  root  formulas  proceed 
as  follows. 

Equation  (1)  is  solved  for  RRxy, 


RRxy  s Rxy  SvSSx 
SSySx 


(4) 


and  RRxy  is  substituted  in  equation  (2), 

2 2 2/  2 2 2 \ 
Sy  (1  - Rxy  ) = SSy  1 - Rxy  3i_|5x„  L 

\ SSy  Sx  / 


Multiplying  the  right  side  through  by  SSy2, 


2 2 2 2 2 2 
Sy  (1  - Rxy  ) = SSy  - Rxy  Sy  SSx 

2 

Sx 


(5) 


(6) 
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and  solving  for  SSy2 , 
2 


2 

, 2 

1 “1 
2 2\ 

= Sy 

(1  - Rxy  ) + 

Rxy  SSx  1 

__ 

Substituting  SSy2  in  equation  (4), 
RRxy  = 


Rxy  £Sx 

£x_. 


V 


1 - Rxy2  + Rxy2(s§x)2 


The  same  method  can  be  used  to  derive  SSz2  and  RRxz  2. 


2 2 

2 

2 

\ 2 

SSz  = Sz 

1 - 

Rxz 

+ Rxz 

SSx) 

V Sxl 

RRxz  = 

Rxz 

•a 

V 


1 - Rxz  + Rxz  /ss^j 


(7) 


(8) 


(9) 

(10) 


. ^ / Iff RRyz  in  6QU3tion/)(3) « W6  3l26br3ic3llv  ch3ru?p 
equation  (2),  dividing  first  by  SSy2  ancl  tlkiSgthe  SJuare  KSf? 


RRxy  ) = V(1  - Rxy  ) 


(11) 


and  dividing  by  SSz2  and  talcing  the  square  root, 


( 1 - Rxz  ) 


(12) 


Substituting  (11)  and  (12)  in  the  denominator  of  (3) 
Rg&.--R*y.Rx3. = — (.RRy.z.-._RRxyRRxzlSSyS.§4.  ___ 


\Z” 1 - Rxy2 


2 , / 2 

Rxy  V 1 - Rxz  SySz 


'f  1 - Rxy2\/l 


Rxz 


(13) 
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and  solving  for  RRyz 


The 

producing 


RRyz  = 
equations  in 


( Rvz  -^RxjrRxz)3y,Sz  + 


assumption  1 can  be 


RRxyRRxz  . (14) 

algebraically  combined, 


2 

RRxyRRxz  = RxyRxz  SvSzSSx 

2 

Sx  SSySSz 


(15) 


Substituting  (15)  in  (14)  and  factoring  out  SySz/SSySSz, 


r 2 

RRyz  = SvSz  Ryz  - RxyRxz  + RxyRxz  SSx_ 
SSySSz  2 

Sx 


(16) 


Formula  (16)  is  the  root  formula  for  the  development  of  the  first  two 
correction  formulas,  and  formula  (14)  serves  as  the  root  formula  for 
correction  formulas  (3)  and  (4).  The  first  correction  formula  is  derived 
on  the  basis  that  neither  SSy  nor  SSz  are  available  and  SSy  and  SSz  are 
estimated  using  the  proportion  SSx/Sx. 

Substituting  the  estimates  for  SSy  (7)  and  SSz  (9)  in  the  root  formula 
(16)  and  simplifying  gives: 


Ryz 

RRyz  = 


RxyRxz  + RxyRxz  S£x_ 
2 

Sx 


(17) 


2 2 2 2 2 2 
- Rxy  + Rxy  SSx^)(1  - Rxz  + Rxz  SSjc^) 

Sx  Sx 

Formula  (17)  is  equivalent  to  Thorndike's  T7  (and  also  to  Gulliksen's 
formula  19,  ref.  4,  p.  149). 

The  second  correction  formula  uses  the  information  (SSy  - Sy),  the 
restriction  of  the  variance  of  te3t  y,  to  estimate  the  restriction  in  SSx 
and  SSz  due  to  selection.  Proceeding  on  this  basis, 

Equation  (2)  is  solved  for  RRxy2,  giving 


RRxy  = 1 - 


2\  2 
(1  - Rxy  ). 


(18) 


Equation  (1)  can  be  expressed  as 


(19) 
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and  substituting  ( 1 8 ) in  (19)  and  solving  for  SSx, 


SSx  = Sx  V_SSv  - Sy  (1-  Rxjr_^. 

SyRxy 


Next,  solve  (2)  for  RRxz,  yielding 


RRxz  s Rxz  SzSSx. 

SxSSz 


Substituting  (21)  in  (2),  solving  for  SSz2and  simplifying 
produces,  J a 


SSz  = Sz 


2 2 2 2 2 2 
L -Rxy  S^_Rxz  + SSy  Rxz 

Sy  Rxy 


^ = r9°Le<’V1SHon  (l6>>  substituting  the 

estimates  for  SSx  (20)  and  SSz  (22)  and  simplifying  produces  the 
second  correction  formula. 

2 2 2 
Rfiyz  = Rxz (SSy  - Sv  ) + RxvRvzSSv 

-v/  2 2 2 2 2 

SSy  Y Rxz  (SSy  - Sy  ) + Sy  Rxy 
Formula  (23)  is  Gulliksen's  formula  G37 . 


§n<*  correction  formulas  employ  the  assumptions  of  the 

first  and  second  correction  formulas,  respectively,  and  make  tne  additional 
assumptions  that  the  new  test  under  consideration,  test  y,  was  administered 
to  the  applicant  group.  Consequently,  there  is  no  need  to  estimate  RRxv, 
SSy  or  SSx,  and  formula  (14)  can  be  utilized  as  the  root  formula. 

Substituting  estimates  for  SSz  (9)  and  RRxz  (10)  used  in 
deriving  the  first  correction  formula  (l7)  in  the  root  formula  (14) 
and  simplifying  gives  the  third  correction  formula, 


RRyz  = __  ..._SyJLR_y:?.--  RxyRxzl 

/ 2/2  2' 
SSyl/  (1  - Rxz  ) + I Rxz  SSx_ 

^ \ Sx 


Rxz  §3x 

-Sx_. 


2/22 
( 1 - Rxz  ) + / Rxz  SSx 

Sx2 
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To  obtain  the  fourth  correction  formula  RRxz  must  be 
derived  in  terms  of  (SSy  - Sy)  by  first  solving  equation  (2) 


for  RRxz2 , 


2 

RRxz 


2 2 
1 - Sz  (1  - Rxz.) . 
2 

SSz 


(25) 


Substituting  (22)  in  (25),  multiplying  and  simplifying  yields. 


RRxz 


Rxz" 


2 2 2 2 

_SSj[_  -_Sy_  + Sy  Rxv . 

2 2 2 2 2 2 
SSy  Rxz  - Sy  Rxz  + Sy  Rxy 


(26) 


To  form  the  fourth  correction  formula,  (22)  and  (26)  are 
substituted  in  the  root  formula  (14)  and  simplified  giving, 


(27) 


_r  Sy(Ryz  - RxyRxz) 

RRyz  = 


2 2 2 2 2 2 
SSy\  j Sy._Rxy._-.__S y _Rxz  _^+_SSy  _R_xz* 

Sy  Rxy 


+ RRxyRxz 


. 2 2 2 2 

CSSy - Sy  _)_+  _Sy  Rxy  __ 

2 2 2 2 2 2 
SSy  Rxz  - Sy  Rxz  + Sy  Rxy 


To  evaluate  the  effects  of  the  selection  ratio,  RRxy , and  RRyz  on  the 
restricted  Ryz  mathematically,  the  process  employed  above  to  obtain 
unrestricted  parameter  estimates  from  restricted  parameters  was  reversed  to 
obtain  explicit  restricted  parameter  estimates  in  terras  of  unrestricted 
parameters.  The  Ryz's  were  then  calculated  as  a function  of  the  selection 
ratio,  RRxy,  and  RRyz  and  compared  to  the  RRyz  to  determine  their 
respective  effects  on  restriction  in  RRyz. 


Since  the  derivation  of  formulas  for  the  explicit  restricted 
parameters  follows  a set  pattern  parallel  to  the  steps  in  deriving  the 
correction  formulas,  the  pattern  will  be  demonstrated  and  the  remaining 
formulas  will  simply  be  given.  This  is  done  for  the  two  cases  employing 
the  assumptions:  (1)  (SSx/Sx)  is  used  to  estimate  the  amount  of 
restriction  as  in  correction  formulas  T7  and  B1  (hereafter  referred  to  as 
assumption  A-SSx) : and  (ii)  (SSy  - Sy)  is  used  to  estimate  the  amount  of 
restriction  as  in  correction  formulas  G37  and  B2  (hereafter  referred  to  as 
assumption  A-SSy) . 


For  A-SSx,  (SSx/Sx), 

Equation  (l)  is  solved  for  Rxy, 


Rxy  = 


(28) 


Rxy  is  substituted  into  equation  (2),  and  multiplying  through 
and  solving  for  Sy  2, 


2 2 
Sy  = SSy  (1 


2 2 
RRxy  ) RRxy 


(29) 


8 


I 


Substituting  Sy2  (29)  in  equation  (28), 


Rxy  = 


_Sx 

EB*y-SSx. 


2 2 _Sx 

! ( 1 - RRxy  ) + RRxy  2 

SSx 


The  pattern  that  parallels  the  development  of  the  correction  formula* 
can  be  noted  by  comparing  (28),  (29),  and  (30)  to  (4)?  (?) and (8)  The 
« Br  3CH  correlations  In  (45!  7), ’and  (dT  become  unriitriStJd1 

(Sx/SSx)1  Thpn*i™  an?  P°  ’ aPd  the  rati0  of  (SSx/Sx)  becomes 

l i>x/SSxj . The  same  pattern  exists  in  the  remaining  derivations  for  s?  and 
Rxz.  Consequently,  these  explicit  equations  can  be  given  as? 


2 2[  2 / 2 
Sz  s SSz  1(1  - RRxz  ) + RRxz/Sx 


_ RRxz  Sx 

««  = 

l/  2 

W ( 1 - RRxz  ) + RRxz 


2)  + RRxz2"5*? 


To  obtain  Ryz,  root  formula  (16)  is  solved  for  Ryz, 


•Z  RRyz  - RRxyRRxz  + RRxyRRxz  Sx^  , 

SSx  . 


to  produce/295  3nd  (31)  are  substituted  into  (33)  and  simplified 


Ryz  = 


RRyz  - RRxyRRxz  ♦ RRxyRRxz  Sx^ 

_SSX_. 


, 2 2/  2\  2 2/  21 

( 1 - RRxy  ) + RRxy  f _&*._)  ^ ( 1 - RRxz  ) RRxz  / 

\ 2 \|  12 

\SSx  / v Vssx  , 


For  A-SSy,  (SSy  - Sy), 

equation  (2)  is  solved  for  Rxy 2,  giving 


2 2 2 
Rxy  = 1 - SSjr  (1  - RRxy  ). 
2 
Sy 
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i 
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The  equations  in  assumption  1 can  be  expressed  as 


_5X  = -fixySy., 
SSx  RRxySSy 


(36) 


and  substituting  (35)  in  (36)  and  solving  for  Sx, 


Sx  = SSx  v Sv  - s 


2 


(37) 


By  comparing  (35),  (36),  and  (37)  to  (18),  (IQ),  and  (20)  the  pattern 
emerges.  The  restricted  correlations  in  (18),  (19),  and  (20)  become 
unrestricted  in  (35),  36).  and  (37)  and  (Sy/SSy)  and  (SSy  - Sy)  become 

(SSy/Sy)  and  (Sy  - SSy).  Applying  this  pattern,  Sz,  Rxz,  and  Rxy  is  given 
as, 


.2  2 2 2 2 2 

Sz  = SSzl  I SSy  RRxy  - SS^  RRxz + Sy  RRxz  J 

SSy  RRxy 


Rxz  s RRxz'1 


2 2 2 2 

Sy - SSy  + SSv  RRxv  _,  and 

2 2 2 2 2 2 


Sy  RRxz  - SSy  RRxz  + SSy  RRxy 


Rxy  a -\/l  - /ssy2\(1  - RRxy2). 
Sy 


(38) 


(39) 


(40) 


v obtain  Ryz,  root  formula  (14)  is  solved  for  Rvz  after 

substituting  values  from  formula  (1),  auiveu  ityz,  arter 

2 


SSySSz 

Ryz  = 

SySz 


( RRyz  - RRxyRRxz)  + RRxyRRxz  Sx 


SSx 


(41) 


Substituting  (37)  and  (38)  in  (41)  and  simplifying  produces, 


2 2 2 
Ryz  = _ -§Sy__)._-t  RRxyRRyzS§y_. 


/ 2 , 2 2 2 2 
Sy  /RRxz  ( Sy  - SSy  ) + SSy  RRxy 


(42) 
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To  examine  the  effects  that  selection  ratio,  RRxy,  and  RRvz  had  on  the 
restricted  Ryz,  the  ratio  (Sx/SSx)  was  assigned  values  of  3,  .5,  and  8 
and  olZ  was  computed  while  varying  RRxy  from  .01  to  1.0  at  76 1 intervals 
"RyLrlues  of  ***’  ?nd  ,To  in3ure  that  each  (Sx/SSx)  and 

(Sy(i,Si,«3*8^1.1^1.5SS0LiS  f0™ulls  for  R« 


RRxy  _SX 

_S_Sx 


2 2 2 
( 1 - RRxy  ) + RRxy  Sx 

SSx 


2 2 
1 - SSxJl  - RRxy  ) 

Sy 


and  the  equation  solved  for  Sy  , 


2 2 2 2 2 
Sy  = SSy  (1  - RRxy  ) + RRxy  _Sx? 

SSx 


(44) 


You  may  notice  that  formula  (44)  is  the  same  as  formula  (29)  even  though 
they  were  derived  from  different  root  equations.  SSy 2 will  be  arbitrarily 
set  at  a constant  20  and  Sy2  will  be  solved  for  Sx/SSx  ratios  of  .2, 

.5,  and  .8. 

A demonstration  of  the  characteristics  of  the  four  correction  formulas 
in  terms  of  more  refined  influences  was  also  performed  by  using  Monte  Carlo 
techniques.  The  Monte  Carlo  study  examined  the  comparative  accuracy  of  the 
four  correction  formulas  as  a function  of  (i)  the  selection  ratio,  (ii) 
RRxy,  and  (iii)  RRyz. 

In  order  to  generate  data  of  known  means,  standard  deviations,  and 
intercorrelations,  a program  (MNRNC)  (2)  (see  Appendix  A)  was  modified  by 
the  authors  and  used.  The  program  uses  the  Marsaglia’s  reasonably  fast 
method  to  generate  normally  distributed  variables  whose  covariances  are 
those  required  by  a specified  correlation  matrix  input  into  the  program. 
Table  1 contains  the  relevant  portion  of  the  correlation  matrix  input  into 
this  program. 


Table 

1 . 

Relevant  Correlations 

Input  Into  MNRNG 

1 

1 2 3 

4 

5 

6 

7 

8 

9 

10 

11 

0.6  0.3 

0.3 

0.R 

0.3 

0.3 

0.3 

0.2 

0.4 

0.5 

2 

0.1 

0.2 

0.3 

0.4 

0.5 

X 

X 

X 

X 

3 

0.1 

0.2 

0.3 

0.4 

0.5 

0.1 

0.1 

0.1 

4 

X 

X 

X 

X 

0.2 

0.2 

0.2 

5 

X 

X 

X 

0.3 

o-3 

0.3 

6 

X 

X 

0.4 

0.4 

0.4 

1 

X 

0.5 

0.5 

0.5 

0 

X 

X 

X 

1§ 

X 

X 

X 

1 The 

correlations 

denoted  by 

X were  not 

used  in 

the  analysis. 
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For  the  purpose  of  this  analysis  variable  1 was  defined  as  variable  x,  and 
variables  2,3,9.10,  and  11  alternated  as  variable  y and  variables 
3 » 4 , 5, 6, 7,  and  8 were  used  for  variable  z.  The  unrestricted  correlation  of 
x and  z (RRxz)  was  a constant  0.30,  the  unrestricted  correlation  of  x and  y 
( RRxy ) ranged  from  0.2  to  0.6  in  increments  of  0.1,  and  the  unrestricted 
correlation  of  y and  z ( RRyz)  ranged  from  0.1  to  0.5  in  increments  of  0.1 
also.  All  possible  combinations  of  RRxy  and  RRyz  were  generated  by  using 
the  various  variables  from  the  generated  data  as  shown  in  Table  2. 

Table  2.  Variables  Used  as  x,y,  and  z for  Assigned  Values  of  RRxy  and  RRyz 


Values  of  RRyz 

0.1 

0.2 

0.3 

0.4 

0.5 

Var  9 

Var  9 

Var  # 

Var  # 

Var 

x = 

1 

1 

1 

1 

1 

0.2 

y = 

9 

? 

q 

9 

9 

z = 

3 

4 

5 

6 

7 

X = 

1 

1 

1 

1 

1 

0.3 

y = 

3 

3 

3 

3 

3 

z = 

4 

5 

6 

7 

8 

Values 

X = 

1 

1 

1 

1 

1 

of 

0.4 

y = 

10 

10 

10 

10 

10 

RRxy 

z = 

3 

4 

5 

6 

7 

X = 

1 

1 

1 

1 

1 

0.5 

y = 

11 

11 

11 

11 

11 

z = 

3 

4 

5 

6 

7 

X = 

1 

1 

1 

1 

1 

D .6 

y = 

2 

2 

2 

2 

2 

z = 

3 

4 

5 

6 

7 

1 Variable  9 used  for  x,  y,  or  z. 


After  a sample  of  1 ,000  subjects  had  been  generated  by  using  the 
correlation  matrix  specified  in  Table  1,  the  sample  was  sorted  into 
descending  order  based  on  variable  1,  the  x variable.  Using  a program 
(REST)  developed  by  Lewis  and  Boone  (see  Appendix  A),  the  sample  was  then 
restricted  on  variable  1 using  five  different  ratios,  1 0% , 20J , 30J,  40J, 
and  50t . For  each  selection  ratio  the  four  formulas  for  correction  for 
restriction  in  range  were  used  to  estimate  the  value  of  RRyz.  This  was 
done  for  each  selection  ratio  for  all  25  combinations  of  RRxy  and  RRyz 
described  in  Table  2 J The  correlations  computed  from  the  restricted  sample 
and  the  unrestricted  sample  were  input  into  a subroutine  (COREST)  developed 
by  Lewis  and  Boone  (see  Appendix  B)  which  employs  all  four  correction 
formulas  and  transformed  tne  estimate  of  RRyz  as  well  as  the  actual  values 
of  RRxy,  RRxz,  and  RRyz  by  using  the  Fischer  R to  Z so  that  the  valuta 
could  later  be  averaged.  This  was  repeated  for  100  samples.  A sumnk..'y  of 
the  process  is  as  follows  in  Table  3. 
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Table  3. 


Summary  of  Processes  Used  in  Study 


1.  Generate  1,000  subjects  with  scores  on  11  variables  as  defined 
by  means,  standard  deviations,  and  correlations. 

2.  Sort  sample  into  descending  order  based  on  scores  on  variable  1. 

3.  Restrict  sample  based  on  selection  ratios  of  1 0% , 20*,  30)1,  40#, 
and  50*. 

4.  Calculate  the  four  different  estimates  of  RRyz  for  each  restricted 
sample  based  on  values  of  HRxy  ranging  from  0.2  to  0.6  and  on 
values  of  RRyz  ranging  from  0.1  to  0.5. 

5.  Transform  all  correlations  and  estimated  correlations  by  using 
Fischer  R to  Z transformation  and  for  use  in  later  averaging. 

The  results  were  then  prepared  in  tabular  and  graphical  form.  Since  the 
sample  size  was  100,000,  significance  tests  were  deemed  inappropriate. 

Results. 

Figures  2 through  7 represent  the  calculated  value  of  the  restricted 
correlation,  Ryz,  when  the  unrestricted  correlations,  RRxz  and  RRyz,  are 
equated  and  assigned  values  of  .2,  .4,  or  .6.  For  each  figure  the 
unrestricted  correlation  RRxy  was  allowed  to  vary  from  .01  to  1.00  by 
increments  of  .01.  The  ratio  of  the  variances  on  the  explicit  selection 
variable  was  assigned  values  of  .2,  .5,  and  .8  and  Ryz  was  plotted  as  a 
function  of  RRxy  for  each  selection  ratio.  This  was  done  for  the  variance 
assumptions  of  T7  and  B1  (A-SSx)  and  also  for  the  assumptions  of  G37  and  B2 
(A-SSy)  for  each  assigned  value  of  RRxz  and  RRyz. 

The  remaining  figures  and  tables  in  the  present  study  are  based  on  the 
data  obtained  through  the  Monte  Carlo  technique  described  in  Table  3-  The 
actual  correlation  matrix  obtained  from  the  Input  of  the  matrix  in  Table  1 
is  contained  in  Table  4. 


Table  4.  Actual  Correlation  Matrix 


1 

2 

\ 

5 

9 

10 

11 


1 

X 


4 

5 

6 

7 

3 

q 

10 

11 

• 30 

• 30 

•3° 

• 30 

.30 

.20 

.41 

.50 

.21 

• 31 

.41 

• 5° 

X 

X 

X 

X 

.09 

.20 

.29 

.40 

.50 

.10 

.09 

.11 

X 

X 

X 

X 

X 

.21 

.21 

.20 

X 

X 

X 

X 

.29 

.31 

.31 

X 

X 

X 

.41 

.40 

.40 

X 

X 

.49 

.49 

.50 

X 

X 

X 

X 

X 

X 

X 

X X 

X X 

X 


In  order  to  assess  the  accuracy  of  prediction  of  each  correlation 
procedure,  an  error  term  was  calculated  based  on  the  absolute  value  of  the 
difference  between  the  actual  unrestricted  correlation  RRyz  and  the 
estimated  correlation  Ryz.  Table  6 contains  this  error  term,  RRyz  - Ryz, 
for  each  correction  formula,  for  each  selection  ratio,  for  each  value  of 
RRxy,  and  for  each  value  of  RRyz.  Figure  8 represents  this  error  term  as  a 
function  of  selection  ratio  for  the  four  correction  formulas  and  for  the 
actual  restricted  correlation  Ryz.  Figure  9 represents  the  error  term  as  a 
function  of  RRxy  for  the  four  formulas  and  Ryz.  Figure  10  represents  the 
error  term  as  a function  of  RRyz  for  the  four  formulas  and  Ryz. 
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SELECTION  RATIO 


Figure  8.  Error  by  selection  ratio  for  the  four 

correction  formulas  and  the  actual  restricted 
value  of  Ryz. 


- 


20 


| RRyz  -Ryz  | 


Figure  9.  Error  by  values  of 
RRxy  for  the  four 
correction  formulas 
and  the  actual 
restricted  value  of  Ryz. 


Figure  10.  Error  values  of  RRyz 
for  the  four  correc- 
tion formulas  and  the 
actual  restricted 
value  of  Ryz. 


Table  5.  Average  Error  in  Estimation  of  RRyz 
Error  by  Formula 


B1 

T7 

G37 

B2 

Means 

Stds 

= 

0.047 

0.04 

0.051 

0.05 

0.073 

0.11 

0.058 

0.07 

Error 

by  Selection  Ratio 

10* 

20 % 

30)1 

40)1 

50$ 

Means 

Stds 

= 

0.112 

0.26 

0.065 

0.14 

0.050 

0.11 

°6°U 

0.028 

0.06 

Error 

by  RRxy 

.60 

.20 

• 30 

.40 

.50 

Means 

Stds 

s 

0.059 

0.16 

0.059 

0.14 

0.058 

0.14 

0.054 

0.14 

0.062 

0.18 

Error 

by  RRyz 

.10 

.20 

• 30 

.40 

.50 

Means 

Stds 

s 

s 

0.073 

0.17 

0.055 

0.13 

0.048 

0.11 

0.060 

0.15 

0.056 

0.18 

21 


Table  6 contains  the  average  Ryz  for  formula  B1  for  each  value  of  RRyz 
by  values  of  RRxy  by  selection  ratio.  To  average  the  correlations,  they 
were  transformed  using  the  Fischer  R to  Z transformation,  averaged,  and 
then  transformed  back  to  a correlation.  Table  7 contains  the  same 
information  for  formula  T7 , Table  8 contains  the  information  for  formula 
G37,  and  Table  9 contains  the  information  for  formula  B2. 


Table  6 

Average  Ryz  for  Formula  B1 
by  RRxy  and  Selection  Ratio  for  Each  RRyz 


RRyz  =0.1 


RRxy  = 

10$  Selection 
20$  Selection 
30$  Selection 
40$  Selection 
50$  Selection 


0.2 

0.082 

0.3 

0.121 

0.4 

0.097 

0 .' 
O.li 

u 

0.6 

0.089 

0.115 

0.133 

0.059 

0.0< 

57 

0.057 

0.098 

0.103 

0.119 

0.058 

0.1 

19 

0.111 

0.110 

0.103 

0.1; 

39 

0.145 

0.092 

0.094 

0.124 

0.1 

37 

0.143 

RRyz  = 0.2 


RRxy  = 0.2 

10$  Selection  0.207 

20$  Selection  0.184 

30$  Selection 


°-3 

0.180 

0.144 

0.157 


0.4 

0.244 

0.232 

0.212 


0.5 

0.215 

0.212 


40$ 

Selection 

0.185 

0.180 

0.203 

0.213 

0.224 

50$  Selection 

RRyz  =0.3 

0.194 

0.189 

0.183 

0.191 

0.207 

10$ 

RRxy  = 

0.2 

0-3 

0.4 

0.5 

0.6 

Selection 

0.272 

0.249 

0.302 

0.277 

0.275 

20$ 

Selection 

0.251 

0.247 

0.290 

0.267 

0.291 

30$ 

Selection 

0.258 

0.292 

0.284 

0.269 

0.287 

40$ 

50$ 

Selection 

0.279 

0.310 

O.306 

0.290 

0. 

301 

Selection 

0.281 

0.307 

0.306 

0.296 

0. 

312 

RRyz  =0.4 

RRxy  = 
10$  Selection 
20$  Selection 
30$  Selection 
40$  Selection 
50$  Selection 


0.2 

0.302 

0.357 

O.368 

0.375 

0.396 


0.3 

0.4 

0.296 

0.388 

O.343 

O.383 

8:118 

O'.  ^5 

0.391 

0.404 

RRyz  = 0.5 

RRxy  = 

10$  Selection 
20$  Selection 
30$  Selection 
40$  Selection 
50$  Selection 


0.4^3 

0.491 

0.509 

0.511 

0.500 


0.4 

O.385 

0.451 

C.453 

0.470 

0.482 


0 4t 
0.482 
0.491 


0.6 

8:211 

0.481 

0.506 

0.504 
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Table  7 

Average  Ryz  for  Formula  T7 
by  RRxy  and  Selection  Ratio  for  Each  RRyz 


RRyz  =0.1 


10H 

RRxy  = 

0.2 

0.3 

0.4 

0.5 

Selection 

0.071 

0.107 

0.123 

0.122 

20H 

Selection 

0.096 

0.107 

0.044 

0.094 

30H 
40 1 

Selection 

0.100 

0.106 

0.051 

0.122 

Selection 

0.104 

0.120 

0.099 

0.148 

50  H 

Selection 

0.087 

0.105 

0.115 

0.137 

0.6 
0.039 
0.039 
0.093 
0 . 1 42 
0.1115 


RRyz  = 0.2 

RRxy  = 0.2 

10H  Selection  0.223 

20H  Selection  0-169 

30$  Selection  0.178 

40H  Selection  0.185 

5011  Selection  0.191 

RRyz  =0.3 


0.3 

0.190 

0.131 

0.159 

0.192 

0.201 


0.4 

0.264 

0.234 

0.213 

0.207 

0.183 


0.5 

0.146 

0.205 

0.195 

0.219 

0.191 


0.6 

0.211 

0.217 

0.197 

0.225 

0.210 


RRxy  = 

0.2 

0.3 

0.4 

0-5 

0.6 

10H 

Selection 

0.254 

0.256 

0.346 

0.285 

0.273 

20  H 

Selection 

0.243 

0.242 

0.309 

0.269 

0.294 

30H 

Selection 

0.26? 

0.292 

0.291 

0.276 

0.287 

4 OH  Selection 

0.278 

0.321 

0.313 

0.299 

0.303 

5 OH 

Selection 

0.279 

0.319 

0.309 

0.297 

0.315 

RRyz  = 0.4 


RRxy  = 
1 0%  Selection 
20H  Selection 
30 H Selection 
4 OH  Selection 
5 OH  Selection 


RRyz  = 0.5 


RRxy  = 
10H  Selection 
20H  Selection 
30H  Selection 
4 OH  Selection 
50H  Selection 


0.2 

g:|S 

0.376 

0.373 

0.396 


0.2„ 
0.484 
0.4; 
0.4f 
0.479 
0.479 


0.3 

0.336 

0.340 

0.377 

0.397 

0.403 


0.4 

0.424 

0.401 

0.406 

0.414 

0.408 


0.4 

0.458 

0.470 

0.467 

0.48 

0.48 


0.5 

8:1S§ 

0.351 

0.367 

0.377 


0.5 

0.500 

0.474 

0.480 

0.492 

0.492 


0.6 

8:  Hi 

0.385 

0.407 

0.411 


0.6 

0.441 

0.476 

0.484 

0.511 

0.507 
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Table  8 


Average  Ryz  for  Formula  037 
by  RRxy  and  Selection  Hatio  for  Each  RRyz 


RRyz  =0.1 

RRxy  = 
10$  Selection 
20$  Selection 
30$  Selection 
40$  Selection 
50$  Selection 

RRyz  = 0.2 


RRyz  =0.3 


RRxy  = 
10$  Selection 
20$  Selection 
30$  Selection 
40$  Selection 
50$  Selection 


RRyz  =0.4 

RRxy  = 
10$  Selection 
20$  Selection 
30$  Selection 
40$  Selection 
50$  Selection 

RRyz  =0.5 

RRxy  = 
10$  Selection 
Selection 
Selection 
Selection 


20$ 
30$ 


0.2 

0.235 

°1?5 


21 

204 

137 


0.3 

0.294 

0.264 

0.183 

0.130 

0.106 


*0$ 

50$  Selection 


0.2 

0.248 

0.323 

O'.  351 

0.352 

0.315 


0.2 

0.228 

0.375 

0.443 

0.436 

0.424 


0.2 

8.-J8 

0.528 

0.519 

0.500 


0.4 

0.154 

0.135 

0.105 

0.145 

0.164 


RRxy  =•  0.2  0-3  0.4 

10$  Selection  0.257  0.223  0-346 

20$  Selection  0.265  0.245  0.309 

30$  Selection  0.268  0.216  0.257 

40$  Selection  0.261  0.192  0.243 

50$  Selection  0.231  0.198  0.221 


0.3 

0.247 

O.332 

0.341 

0.327 

0.315 


0 • 3 

0.261 

0.429 

0.412 

0.402 

0.400 


0 .1*45 

§:li 

0.525 

0.507 


0.4 

0.355 

0.331 

0.317 

0.338 

0.339 


0.4 

0.440 

0.416 

8:S1J 

0.437 


0.4 

0.370 

0.489 

0.486 

0.504 

0.512 


°-5 

0.260 

0.106 

0.141 

0.160 

0.141 


0.5, 

§:!?? 

0.222 

0.233 

0.195 


0.5, 

0.247 

0.267 

0.286 

0.309 

0.301 


0.6 

0.198 

O.O85 

0.104 

O.I58 

0.146 


0.6 

O.334 

0.237 

0.219 

0.233 

0.211 


0.6 

0.326 

0.295 

0.294 

0.310 

0.315 


0.6 

0.409 

0-362 

0.394 

0.412 

0.413 


0.6 

0.342 

0.472 

0.491 

0.514 

0.507 
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Table  9 

Average  Ryz  for  Formula  B2 
by  RRxy  and  Selection  Ratio  for  Each  RRyz 


RRyz  =0.1 

RRxy  = 
10)1  Selection 
2 051  Selection 
30%  Selection 
40%  Selection 
50%  Selection 

RRyz  =0.2 

RRxy  = 
10%  Selection 
20%  Selection 
30%  Selection 
40%  Selection 
50%  Selection 

RRyz  = 0.3 

RRxy  = 
10%  Selection 
20%  Selection 
30%  Selection 
40%  Selection 
50%  Selection 

RRyz  = 0.4 

RRxy  = 
10%  Selection 
20%  Selection 
30%  Selection 
40%  Selection 
50%  Selection 


0.2 

0.137 

0.152 

0.134 

0.140 

0.110 


0.239 

0.193 

8:888 

0.207 


0.2 

0.265 

0.258 

0.279 

0.292 

0.290 


0.2 

8:18? 

8:188 

0.399 


0.199 

0.149 

0.114 

0.093 


0.3 

0.205 

0.198 

0.186 

0.17? 

0.186 


0.3 

0.245 

0.288 

0.311 

0.311 

0.303 


0.3 

0.279 

0.375 

0.382 

0.387 

0.388 


0.4 

0.116 

0.106 

0.066 

0.124 

0.149 


0.4 

0.290 

0.273 

0.235 

0.221 

0.204 


0.4 

0.314 

0.309 

0.300 

0.320 

0.324 


0.4 

8:113 

0.412 

0.419 

0.420 


0.5 

0.221 

0.104 

0.129 

0.145 

0.139 


0.5 

0.298 

0.216 

0.211 

0.220 

0.193 


0.5  , 
0.316 
0.267 
0.277 
0.29b 
0.298 


8:118 


0.6 

0.216 

0.085 

0.105 

0.154 

0.144 


0.6 

0.361 

0.237 

0.216 

0.229 

0.207 


0.330 

0.295 

0.293 

0.306 

0.312 


8:3  ll 
8:28i 

0.409 


RRxy  = 
10%  Selection 
20%  Selection 
30%  Selection 
40%  Selection 
50%  Selection 


0.2 

0.412 

0.386 

0.450 

0.462 

0.477 


0.3 

0.439 

0.502 

0.501 

0.510 

0.497 


0.4 

8:11s 

0.467 

0.483 

0.497 


0.5 

0.395 

0.472 

0.476 

0.467 

0.493 


0.6 

0.295 

0.469 

0.488 

0.50? 

0.504 
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Figures  11  through  30  graphically  represent  the  estimated  RRyz  (Ryz) 
as  a function  of  assigned  values  of  RRxy.  Each  assigned  value  of  RRyz  is 

fraphed  separately  for  each  correction  formula,  and  each  graph  contains  a 
ine  representing  each  of  the  five  selection  ratios.  The  actual  value  of 
RRyz  is  represented  as  a straight  line.  Figures  11  through  15  represent 
Ryz  as  a function  of  RRxy  by  selection  ratio  for  formula  Bl . Figures  16 
through  20  represent  Ryz  as  a function  of  RRxy  by  selection  ratio  for 
formula  T7 • Figures  21  through  25  represent  Ryz  as  a function  of  RRxy  by 
selection  ratio  for  formula  B2,  and  Figures  26  through  30  represent  Ryz  as 
a function  of  RRxy  by  selection  ratio  for  formula  G37. 


RRxy 

Figure  11.  The  estimated  unre- 
stricted correlations 
for  formula  Bl, 

RRyz  * . 1 . 


RRxy 

Figure  12.  The  estimated  unre- 
stricted correlations 
for  formula  Bl, 

RRyz  = . 2 . 


1 0 X Selection 
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Discussion. 

In  any  Monte  Carlo  study  a decision  must  be  made  concerning  which 
components  are  to  be  varied  and  what  the  range  of  their  variation  will  be. 
The  components  selected  for  variation  and  their  range  in  this  study  were 
established  subjectively  based  on  values  the  authors  considered 
representative  of  practical  situations.  Consequently,  the  discussion  of 
the  results  is  more  a comparison  of  the  practical  utility  of  each  formula 
rather  than  a strict  mathematical  comparison. 

Restricted  correlations.  Figures  2 through  7 present  the  estimated 
restricted  correlations  by  selection  ratios  (Sx/SSx)  of  .2,  .5,  and  .8  for 
RRvz  values  of  .2,  .M,  and  .6  across  values  of  RRxy  ranging  from  .01  to 
1.00  in  .01  increments.  As  presented  in  the  figures,  the  estimated  Ryz's 
converge  with  the  actual  RRyz  as  a function  of  RRxy  without  regard  to 
selection  ratio.  As  the  values  of  RRxy  increase  beyond  the  convergent 

Eoint,  Ryz  becomes  more  an  underestimate  of  RRyz,  and  as  RRxy  is  decreased 
elow  the  convergent  point,  Ryz  becomes  more  and  more  an  overestimate  of 
RRyz.  Further,  the  degree  of  error  in  Ryz  sharply  increases  as  the 
selection  ratio  decreases.  It  is  also  apparent  from  the  figures  that  as 
RRxz  and  RRyz  increase,  the  point  of  convergence  for  Ryz  ana  RRyz  on  RRxy 
also  increases,  indicating  the  point  of  convergence  is  related  to  the 
various  intercorrelations  of  RRxz,  RRyz,  and  RRxy,  but  not  to  the  selection 
ratio.  The  only  difference  between  tne  estimates  of  Ryz  when  using  the  two 
different  sets  of  assumptions  to  derive  the  formulas  (A-SSy  and  A-SSx)  is 
the  point  of  convergence.  The  A-SSy  assumptions  result  in  convergence 
lower  on  RRxy  than  the  A-SSx  assumptions. 

The  general  practical  conclusion  to  be  drawn  from  this  portion  of  the 
study  is  that  unless  the  situation  being  studied  contains  the  exact  and 
particular  interrelationship  of  RRxz,  RRyz.  and  RRxy  necessary  for  Ryz  to 
converge  with  RRyz,  the  restricted  Ryz  will  consistently  be  either  an 
underestimate  or  overestimate  of  the  unrestricted  RRyz,  with  the  amount  of 
error  increasing  sharply  as  the  selection  ratio  becomes  more  and  more 
extreme.  Consequently,  correcting  the  restricted  correlations  is  almost 
always  warranted. 

Main  effects.  Table  5 demonstrates  the  overall  accuracy  of  each  of 
the  four  formulas  in  terms  of  the  average  amount  of  error  each  incurred  in 
estimating  RRyz.  Their  rank  order  from  least  to  most  error  is:  B1,  T7, 

B2,  and  G37.  The  first  three  formulas  are  not  remarkably  different: 
however,  G37  is  far  less  accurate  than  B1 , T7 , and  B2.  The  clearest  effect 

on  error  is  produced  by  the  selection  ratio  (Table  5) . As  the  selection 
ratio  becomes  more  extreme,  the  amount  of  error  increases,  with  the 
increase  becoming  larger  and  larger  with  each  step  down  in  the  selection 
ratio.  Table  5 shows  little  fluctuation  in  error  for  RRxy  and  no 
systematic  pattern.  The  effects  of  RRyz  in  Table  5 show  a pattern  that  was 
found  consistently  throughout  the  analyses.  When  RRyz  = RRxz,  the  error 
component  is  at  a minimum.  RRxz  was  held  at  a constant  .30  for  this  study 
and,  as  can  be  noted  in  Table  5,  the  error  increases  as  RRyz  moves  in 
either  direction  from  .30. 

Practical  conclusiorrs  related  to  main  effects  include  the  following. 

If  sufficient  information  is  available,  the  B1  formula  produces  the  most 
accurate  estimate  for  RRyz.  In  order  to  have  sufficient  information  to  use 
B1 , the  new  test  being  evaluated  would  need  to  be  administered  to  the 
applicant  group  at  the  same  time  the  old  selection  test  is  administered. 
Then  RRxy  and  SSy  are  available  for  use  in  B1 . If  the  new  test  being 
evaluated  was  not  administered  to  the  applicant  group,  then  the  most 
accurate  correction  formula  would  be  T7  which  does  not  require  RRxy  and 
SSy. 
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The  selection  ratio,  it  appears,  has  the  largest  impact  on  errors  in 
estimating  RRyz.  If  selection  is  extreme,  10  percent  or  less,  the  formulas 
for  estimating  RRyz  are  unstable  and  highly  inaccurate.  This  is  a 
difficult  practical  situation  to  resolve.  A general  advertisement  for 
applicants  without  sufficient  specific  qualification  statements  results  in 
a larger  number  of  unqualified  candidates  and  more  extreme  selection. 
However,  with  a highly  specific  advertisement  self-selection  becomes  a 
secondary  selection  process,  and  the  statistics  computed  on  the  applicant 
group  are  already  restricted  producing  spuriously  low  validity 
correlations.  One  strategy  would  be  to  administer  the  selection  tests  to  a 
random  sample  in  the  general  population,  stratifying  by  race  and  sex  in 
order  to  meet  Equal  Employment  Opportunity  Commission  requirements.  This 
would  yield  unrestricted  variances  without  the  influence  of  any  selection 
procedure. 

Since  RRyz  is  not  known  and  RRxy  is  computed  after  the  test 
administration,  little  practical  guidance  can  be  offered  related  to  these 
parameters.  The  usual  advice  is  clearly  applicable,  viz,  choose  a test  or 
construct  a test  for  selection  that  parallels  the  actual  job  tasks  as 
closely  as  possible. 

Interaction  effects.  As  seen  in  Figure  8,  when  error  in  prediction  is 
examined  by  selection  ratio  for  each  formula  and  for  the  actual  restricted 
correlation  of  Ryz,  there  is  a tremendous  amount  of  error  for  the 
10-percent  selection  ratio,  with  formula  B1  doing  a much  better  job  than 
either  T7 , B2 , or  G37  in  estimating  RRyz.  As  the  selection  ratio  increases 
beyond  moderate  selection  (30  percent),  the  formulas  tend  to  perform 
similarly  in  estimating  RRyz,  with  the  exception  of  G37  which  consistently 
has  more  error  than  the  other  three  formulas  across  all  selection  ratios. 

Figure  9 demonstrates  that  formula  B1  again  is  consistently  the  better 
estimator  of  RRyz  across  values  of  RRxy.  It  can  also  be  noted  from  Figure 
9 that  as  the  value  of  RRxy  increases,  Ryz  rapidly  becomes  a poorer 
estimator  of  RRyz,  Darticularly  after  it  passes  tne  point  at  which  RRyz 
equals  RRxz  (.30).  Once  again,  G37  is  a much  less  accurate  estimator  of 
RRyz  than  the  other  three  formulas. 

When  RRyz  is  less  than  .30,  as  shown  in  Figure  10,  B1* is  the  better 
estimator  of  RRyz.  All  formulas  converge  when  RRyz  equals  RRxz  (.30)  and 
T7  i3  the  best  estimator  for  higher  values  of  RRyz  although  the  differences 
are  small.  Once  again  formula  G37  is  clearly  the  least  accurate  estimator 
of  RRyz. 

In  terms  of  the  selection  ratio  by  RRxy  by  RRyz  interaction,  Figures 
11  through  30  revealed  the  following.  Error  in  this  case  is  mainly 
influenced  by  the  selection  ratio  with  some  minor  influence  on  error  added 
by  RRxy  and  RRyz.  With  a low  selection  ratio  and  low  RRxy  and  RRyz,  the 
error  component  is  relatively  large.  When  the  selection  ratio  is  low  and 
RRxy  and  RRyz  are  high,  the  errors  are  moderate  to  large.  With  a hign 
selection  ratio  and  low  RRxy  and  RRyz,  the  errors  are  moderate  to  small. 
When  the  selection  ratio  ana  RRxy  and  RRyz  are  all  high,  the  comparative 
error  is  minimal. 

The  four-way  interactions  given  in  Tables  6 through  9 and  Figures  11 
through  30  for  selection  ratio/RRxy/RRyz/formula  show  that  when  tne 
selection  ratio  is  small  to  moderate  (10  to  30  percent)  across  all  values 
of  RRxy  and  Ryz,  formula  B1  results  in  the  least  amount  of  error.  When 
there  is  a high  selection  ratio  and  high  RRxy  and  RRyz,  all  the  formulas 
tend  to  be  about  the  same  in  accuracy. 
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The  practical  implications  for  the  interaction  effects  can  be  stated 
briefly.  The  selection  ratio  has  such  an  overwhelming  effect  that 

generally  the  interaction  effects  are  primarily  due  to  the  selection  ratio. 

hen  the  selection  ratio  is  small  to  moderate  (10  to  30  percent),  formula 
B1  is  clearly  the  most  accurate  estimator  and  should  be  used  regardless  of 
RRxy  and  RRyz.  When  the  selection  ratio  goes  above  30  percent,  B1 , T7 , and 
B2  are  practically  equivalent.  Formula  G3T  is  the  least  desirable 
correction  formula  across  conditions.  Thus,  overall,  B1  results  in  the 
most  accurate  estimates  of  RRyz.  especially  when  the  selection  ratio  is  30 
percent  or  less,  regardless  of  the  values  of  RRxy  or  RRyz. 
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Appendix  A 


PROGRAM  MNRNG 

DIMENSION  R< 11,11), XMM( 11), STD( 1 1 ) , CP( 30 , 30) , C( 30 , 30) . X(  1 1 ) . 

Z(  1 1) ,TV( JO.BOi.TDSC  30) ,XBAR( 1 1 ) ,XM( 11) ,SDEV( 1 1 ) ,STP( 1 1 ) , Y( 1 1 ) , 
XM2(11).VAR(30,30),RC(30,30),VAT(11) 

BYTE  XMAT( 72) . YMATl  72) 

0PEN(UNIT=1  ,NAME= 'MONTY. DAT'  TYPE='NEW') 

OPENl UNIT=2 ,N AM£= 'COR .DAT’ .TYPEs’OLD’ .READONLY) 


[unit=3;name=’corr.dat{) 

T=4,f 


901 

902 


OPENl .. . 

OPEN(UNIT=4,NAME='CNTL.DAT' ,TYPE='OLD' .READONLY) 
ICAL=0 

READ(4.901.END=99)NV 
F0RMAT(20I6) 

READ(4,902)XMAT 
FORMATl 72A1 ) 

READ (4, 902 )YM AT 

“ .END=99)NOS 


21 


; 1 ,NV) 
<1,NV) 


30 


78 

49 


22 


20 

908 

99 


READl 4 , 901 , END=99) NO 
READ(2,XMATHXMM(J), 

READv  2.XMAT) ISTDl J) , 

FORMAT(F6.3) 

DO  30  J=1,NV 
XM(J)=0. 

XM2( J) =0 . 

DO  30  K=1,NV 
CP( J ,K) =0  . 

RC(J.K)=0. 

DO  49  1=1  ,NV 

READ(2iYMAT)(R(I,J),J=1,NV) 

FORMATl <NV>F3 . 0) 

R( I, I) =0 .0 
CONTINUE 
DO  22  1=1, NOS 

CALL  MS  CORE  ( R ,NV  .XMM.STD  ,NV  ,X , 1 ,NV  , 1 ,NV  , 3 , ICAL) 
DO  20  J= 1 ,NV 
XM( J) =XM(J)+X(J) 

XM2( J) =XM2( J)+X( J) ••2 
DO  20  K=1 ,NV 
CP(J,K)=C1 
NRITEM 
FORMATl 
CONTINUE 
DO  110  J=1.NV 
XBAR(J|=XM(J)/NOS 


\ — I >11* 

)=CHJ,K)+X(J)*X(K) 


SDEVIJ 
DO  100  K= 1 


100 

980 


IF 
IF 
RC 
•XM(  J 


J.EQ.K 
J . EQ . K ) GO' 


SQRT( ( ( N0S»XM2 ( J ) ) - ( XM( J ) »»2 ) ) / ( NOS* ( NOS- 1 . 
* NV 


))) 


K)  = 

9 


1. 


J.K)=( (NOS*CP( J.K) )-( XM( J) *XM( K) ) )/SQRT( (N0S«XM2( J)- 
•*2) * (N0S*XM2{ K;-XM(K) **2) ) 

CONTINUE 

WRITE(3,980)SDEV(J).(RC(J,K) ,K=1,NV) 

F0RMAT(F7.2,<NV>F6.3) 

STOP 
END 

SUBROUTINE  MSCORE  ( R.NRR .XMM.STD ,NDRR ,X ,NRRX ,NCCX , 

+NDRRX.NDCCX .NCOUNT .ICAL) 

C PROGRAMMED  BY  PETER  TAM 

C THIS  SUBROUTINE  GENERATES  MULTIDEMENSIONAL  SCOHES.  THE  USER 
C SPECIFIES  THE  NO.  OF  PERSONS  AND  SCORES  FOR  EACH  PERSON.  BOTH  PEOPLE 
C AND  SCORES  CAN  BE  MADE  UNLIMITED  BY  ADJUSTING  THE  DIMENSION  STATEMENTS 
C 

C R*  THE  INPUT  R MATRIX  OF  INTERCORRELATIONS,  THE  DIOGANAL  AND  UPER  DIAGONAL 
C ELEMENTS  SHOULD  BE  0.0,  TYPE  IN  THE  WHOLE  MATRIX  OF  R 
C NRR=  THE  NO.  OF  ROWS  ACTUALLY  IN  R MATRIX 
C XMM=  THE  VECTOR  OF  MEANS  FOR  THE  VARIABLES 
C STD  = THE  VECTOR  OF  STANDARD  DEVIATIONS  FOR  THE  VARIABLES. 

C NDRR=  THE  NO.  OF  ROWS  DIMENSIONED  FOR  R MATRIX  IN  THE  CALLING  PROGRAM 

C X=  THE  MATRIX  OF  SCORES  GENERATED.  ROWS  REPRESENT  PERSONS,  COLUMNS  THE  SCORES 

C NRRX=  THE  NO.  OF  SUBJECTS  (ROWS)  NEEDED 

C NCCXs  THE  NO.  OF  VARIABLES  (COLS)  NEEDED  FOR  EACH  PERSON 

C NDRRXs  THE  NO.  OF  ROWS  DIMENSIONED  FOR  X IN  THE  CALLING  PROGRAM 

C NDCCXa  THE  NUMBER  OF  COLUMNS  DIMENSIONED  FOR  X IN  THE  CALLING  PROGRAM 
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NCOUNT.. THE  PRINT  CHOICE.  IF  VALUE  LESS  THAN  OR  EQ.  3, 
IF  NCOUNT  GT  3 WILL  PRINT  ON  DESIGNATED  NCOUNT. 

DIMENSION  R( NDRR .NDRR ) .XMM( NDRR) , STD(NDRR) , VAR( 30 
+ X(NDRRX.NDCCX) ,TV(30r30) ,TDS(30) 

DO  10  I=1,NRR 
VAR(I.I)=STD(I)*«2 
10  CONTINUE 
NNR=NRR-1 
DO  30  J=1,NNR 
K=J+1 

DO  20  I=K ,NRR 

VAR? I , J) =R( I, J) *STD( J) #STD(  I) 

20  VAR(J.I) =VAR( I , J) 

30  CONTINUE 

I F( NCOUNT . GT . 2) G0T034 
WRITE? NCOUNT, 31) 

31  FORMAT! 1H1 ,///5X , 'VAR-COV  MATRIX  FROM  INPUT  CORREL . 


32 

13 

35 

40 


50 

997 


80 

90 

100 

150 

180 


170 


210 

220 

230 


250 

260 

500 


520 

540 


DO  32  1=1. NRR 
WRITE ( NCOUNT ,33)(VAR(I,J).J=1,NRR) 

FORMAT( ( 1H0 ,5X ,6( F15 .2 ,2X) ) ) 

DO  35  1=1, NRR 
DO  35  J=1.NRR 
C(I,J)=0.6 
DO  40  Isl.NRR 
C(I,1)=VAR(I,1) /STD( 1 ) 

DO  100  1=2, NRR 

IM1=I-1 

A=0.0 

DO  50  J= 1 , IM1 
A=A+C(I.J)**2 
FORMAT(I3,2F10.3) 

C?I,I)=SQRT(VARfl,I)-A) 

IP1=I+1 

IF( IP1 ,GT.NRR)GOT0150 
DO  90  L=IP1 ,NRR 
B=0.0 

DO  80  K=1,IM1 

C(lTi1 =? VAR^L^if-B)/C(I ,1) 

CONTINUE 
CONTINUE 
DO  180  1= 1 ,NRRX 
DO  180  J=1 ,NCCX 
X(I.J)=0.0 
DO  230  Irl.NRRX 
DO  170  L=1,NCCX 
CALL  MIX  (ZW.ICAL) 

TDS( L) =ZW 
DO  220  J=1,NCCX 
DO  210  K=1 , J • 

X?I, J) =X?I , J)+(C( J ,K)*TDS( K) ) 

X? I,J)=X( I, J)+XMM( J) 

CONTINUE 

CONTINUE 

IF(NCOUNT.GE. 3) RETURN 
WRITE( NCOUNT ,250) 

F0RMAT(///5X  ‘TRANSFORMATION  MATRIX  C» , //) 

DO  260  1=1, NRR 

WRITE? NCOUNT ,33)(C(I,J),J=1,NRR) 

WRITER NCOUNT ,500) 

FORMAT (//5X.‘ RECHECK  PARAMETER  INFORMATION' ,//5X , 
•ORIGINAL  PARAMETER  R MATRIX') 

DO  520  1=1, NRR 
WRITE( NCOUNT,' 

WRITE?" 


NCOUNT, 33)(R(I,J).J=1lNRR) 

NCOUNT, 540)  ( XMM(  I j J.=  \ ,NRR) 

FORMgT^/|X^‘^IGINAL  VECTOR  OF  M 


MEANS' ,/ 


i ia.uvtu.c iCa; / 

WRITE! NCOUNT, 300) 


NO  PRINT. 
,30),C(30,30), 


MATRIX') 
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300 


FORMAT* 1H1 ,///20X  'MULTIDIMENSIONAL  SCORES' ,/15X, 

1 'NOTE. .ROWS  ARE  SUBJECTS,  COLS  ARE  VARIABLES') 

RETURN 

END 

SUBROUTINE  MIX(AA.IC) 

DOUBLE  PRECISION  ILT 
BYTE  NUM(IO) ,SEC(0) 

HANDOM  NORMAL  DEVIATES  BY  MARSAGLIA-S  REASONABLY  FAST  METHOD. 

PROGRAMMED  BY  ONE  OF  MEETER-S  STUDENTS.  1970 

DATA  A/ . 8638/ , B/ . 1 107/ . C/ . 0228002039/ , 0/ . 0026997961/ 
DATA  A0/17.49731196/,Al/2.36735l63/.Ai/2. 15737544/ 

DATA  NUM/'O' ,'1',*2' , '3' , '4' , '5' ,f6' , '7' , '8' , '9' / 

IC=IC+ 1 

IF(IC.NE. 1)G0T01 
ILT=0 

CALL  TIME  (SEC) 

DO  56  J=1,10 

IF( SEC(7) . EQ .NUM( J) )ILT=ILT+( J— 1 ) * 1 0 


56 


98 

1 


4 

21 

19 


9 

11 

12 

14 
6 

23 

15 


17 

18 

22 

16 


) .EQ.NUM 
) .EQ.NUM 


ILTsILT+J-1 
ILT=ILT+( J-1 ) *60 


IF1SEC18 
IF(SEC(5 
CONTINUE 
11=0 
12=0 

DO  98  K=1.ILT 
XYZ=RAN(I1 ,12) 

U1=RAN( II , 12) 

IF(U1 .GT. A)G0T02 
AA=2 .*(U1/A+RAN( II , 

RETURN 
IF(U1 .GT. A+B)G0T04 
AA=1 . 5#( ( U1-A) /B+RAN( II ,I2)-1 . ) 
RETURN 

IF(U1  .GT.  A+B+OG0T06 
X=6.*(U1-A-B)/C-^. 


I2)+RAN(I1 ,I2)-1 .5) 


IrAl 


.8*ran(ii,i25 

BS(  X) 


IF( XA .GT. 1 . )G0T07 

G3=A0»EXP(-(X»X)/2.)-A1»(3.-XA)*»2 

G0T013 

IF(XA.GT.1 .5)G0T09 

G3=A0"EXP(-(X»X)/2.)-A1*(3.-XA)»*2-A2»(1.5-XA) 

G0T013 

IF( XA .GT. 3 . )G0T012 
G3=A0"EXP(-(X»X)/2.)-A1«(3.-XA)»»2 
GOTO 13 
G3=0 . 

IF(Y.LT.G3)G0T014 

X=6.«RAN(I1 ,I2)-3 

G0T021 

AA=X 

RETURN 

V1=2.*( ( U 1 — A— B— C) /D) — 1 . 

V2=2.#RAN( II ,12) — 1 . 

R=V1*VHV2«V2 
IF(R5.GT.1.)G0T022 
Z=SQRT( (9.-2. *ALOG( R) )/R) 

X1=V1»Z 

X2=V2,Z 

XIAsABS(XI) 

IF(X1A.GT.3.)G0T016 

X2A=ABS(X2) 

IF( X2A .GT. 3 . )G0T022 

AA.X2 

RETURN 

VI =2  .*RAN( II, 12)  — 1 . 

G0T023 

AAsXI 

RETURN 

END 
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PROGRAM  REST 

THIS  PROGRAM  TAKES  SORTED  OUTPUT  FROM  FILE  NORM.DAT 
AND  CORRELATIONS  AND  SD'S  FROM  COR. DAT  AND  RESTRICTS 
THE  FILE  BY  INCREMENTS  OF  1056 
THE  PROGRAM  CALLS  A SUBROUTINE  (COREST)  WHICH 
CORRECTS  FOR  RESTRICTION  IN  RANGE  USING  FORMULAS 
B-1 , B-2 , G-37,  AND  T-7 

DIMENSION  X( 1 1 ) ,S( 1 1 ) , R( 1 1 . 1 1 ) ,SX( 1 1 ) ,SX2( 11), SXY( 1 1 , 1 1 ) ,C(4) , 
• RR(  11.1 1)  ,SS(  1 1 ) ,IV(5)  ,IR(5l 

OPEN( UNIT= 1 .NAME: ' NORM . DAT' .TYPE: ' OLD'  READONLY) 

THIS  FILE  CONTAINS  THE  DATA  GENERATED  BY  THE  PROGRAM  MNRNG 
0PEN(UNIT:2 .NAME: 'CORR . DAT' .TYPE: ' OLD'  READONLY) 

THIS  FILE  CONTAINS  THE  UNRESTRICTED  CORRELATIONS  AND  SD'S  FROM  NORM 
OPEN( UNIT:3 .NAME: ’ RNEW . DAT’ ) 

XT=0 . 


3 

910 


920 


10 


19 

20 


25 


99 


:2 

9 


Jj:. 

ho 

IR  5)=11 


IRi 

IRi 

IRi 

IRi 


IVI 
IV 
IV  i 
IVI 


31: 


;1 

=4 

3 


IV(5) =3 
DO  3 J: 1.11 

READ(2,9^0)SS( J) , (RR( J.K) ,K:1 , 1 1 ) 

FORMAT(F7.2,11F6.3) 

DO  1 Jal.11 

IWlo0' 

SX2( J):0. 

DO  1 K: 1,11 
R(J,K)=0. 

SX  y(  J ,K)  :0 . 

DO  60  IRS: 1.5 

DO  10  MR=1 , lOO 

READ(1,920)(X(J),J:1,11) 

F0RMAT(11F11.6) 

XT:XT+1. 

DO  10  Jrl.11 
SX(J):SX(J)+X(J) 

SX2(J):SX2(JHX(J)»»2 
DO  10  Ksl.11 

SXY(J,K):SXY(J,K)+X(J)*X(K) 

S?J)=SQRT(||xT»SX2(J))-(SX(J)»»2))/(XT»(XT-1 .))) 

SQRt(  ^X^SX2?J)-Si(  j]*^2)  WxT*Sx!?Ik  j-SX(  K)  **2) ) 

CONTINUE 

CONTINUE 

Jsl 

DO  25  IM= 1 ,5 
KsIR(IM) 

L=IV( IM) 

MNsL+4 

DO  25  NsL.MN 

YZsRR(K.N) 

XYsRR(J.K) 

ORsS(N) 

CALL  COHEST( YZ,XY,SS( J) ,SS(K) ,0R ,S( J) , S( K) , R( J , K) ,R(J ,N) ,R(K,N) 
ZB— H ( K.N) 

WRIT£( 3,925)0 ,YZ,RR( JfN) f XY,SS( J) ,S(  J) ,SS(K) ,S(K) ,ZB, 

CONTINUE 

STOP 

END 


,XT,C) 
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Appendix  B 


SUBROUTINE  COREST 

SUBROUTINE  COREST( RRYZ ,RRXY ,SXX ,SYY ,SZ ,SX ,SY , RXY  ,RXZ ,RYZ ,XT,C) 
C 
C 

C THIS  SUBROUTINE  TAKES  INPUT  FROM  THE  PROGRAM  REST  AND 
C CALCULATES  ESTIMATED  RRYZ  USING  FORMULAS  B-1 , B-2,  AND 
C G-37,  AND  T-7 

C RRYZ. RRXY=  UNRESTRICTED  CORRELATIONS;  SXX,SYY=UNRESTRICTED  SD'S 
C RXY,RXZ,RYZ=  RESTRICTED  CORRELATIONS,  SX ,SY ,SZ=RESTRICTED  SD'S 
C X=  EXPLICIT  SELECTION  VARIABLE,  Y=  IMPLICIT  SELECTION  VARIABLE 
C Z=  CRITERION  VARIABLE 

C THE  ESTIMATES  ARE  TRANSFORMED  USING  FISCHER  R TO  Z SO  THAT 
C THEY  CAN  LATER  BE  AVERAGED  AND  RETURNED  TO  PROGRAM  REST 
C 
C 

DIMENSION  C(4) 

910  FORMAT( 10F7.3) 

DO  478  J=  1 , 4 
C(J)=0. 

478  CONTINUE 

XX=(SXX*#2)/(SX*«2) 

XXX-SQRT( XX) 

FIR=(SY»(RYZ-(RXY*RXZ)))/(SYY«SQRT((1-RXZ«*2)+ 

1 ( RXZ#*2#XX) ) ) 

SEC=( RXZ*XXX) /SQRT( ( 1-HXZ**2)+( RXZ**2*XX) ) 

B1=FIR+SEC*RRXY 

DET=RYZ-RXY*RXZ+RXY»RXZ»XX 

RM=RXY»»2 

XNUM=SQRT( ( ( 1 .-RM)+( RM«XX) )•( ( 1 ,-RXZ«»2)+( RXZ«»2»XX) ) ) 

T7=DET/XNUM 

SB=SY**2 

DEN=RXZ»(SYY»«2-SBURXY«RYZ»SB 

XNUMsSY Y*SQRT( ( RXZ**2# ( SYY**2-SB ) )+SB»(RXY»»2) ) 

0X=SYY,#2 

G37sDEN/XNUM 

FIR= ( RYZ-RXY»RXZ )»SY«SZ 

SEC=SYY#SQRT( SZ**2* ( ( (SY##2*RXY**2) -( SY**2I,RXZ#,2) ♦ 

• (SYY»»2«RXZ«»2) )/(SY»»2»RXY»«2) ) ) 

THIRsFI R/SfiC 

F0UR=RXZ»SQRT((SYY<»»2-SY»»2+(SY«*2«RXY*»2))/((SYY«»2,» 

• RXZ»»2)-(SY«»2»RXZ»»2)+(SY«»2»RXY»»2) ) ) 

R*RRXY 

(1. 

(1. 

(1. 

(1. 

99  RETURN 
END 
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♦G37)/(1.-G37)))/2. 


mm 


B2=THIR-*-F0U 
C(3)=(AL0G( 
C(4)=(AL0G( 
C(1)=(AL0GI 
C(2)  =(  ALOG( 


